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SPECIFICATION 



1. TITLE OF THE INVENTION 

A component-based method for face description and recognition using high-order 
eigencomponents 

2. WHAT IS CLAIMED IS 



(1) A method for extracting component features for face description, comprising the 
steps of: 

calculating the 1 st -order eigencomponents with training face images; 
calculating the 2 nd -order eigencomponents with the said training face images; 
getting the 2 nd -order component features for a face image to be described from the 
2 nd -order eigencomponents; and 

selecting features to describe the said face image from the 2 nd -order component 
features. 

(2) A method for extracting component features for face description, comprising the 
steps of: 

calculating the I s -order eigencomponents with training face images; 

getting the l rt -order component features for a face image to be described from the 1* 

-order eigencomponents; 

calculating the 2 nd -order eigencomponents with the said training face images; 
getting the 2 nd -order component features for the said face image to be described from 
the 2 nd -order eigencomponents; and 

selecting features to describe the said face image from die l rt -order and 2 nd -order 
component features. 

(3) A method for describing a face image, comprising the steps of: 

scanning the said facial component (such as eyes, eyebrows, nose, mouth and 
outline) into a single dimensional array of pixels; 

subtracting the said single dimensional array of pixels with the average component; 
multiplying the said subtracted single dimensional array of pixels with the said 1 st - 
order and high-order eigencomponents (such as eigeneyes, eigeneyebrows, 
eigennoses, eigenmouths and eigenoutlines); 

using the resulting component features as description of the component; and 
combining the various component features (such as eyes features, eyebrows features, 
nose features, mouth features and outline features) using different attention weights 
to describe the face. 

(4) A method for describing a face image, comprising the steps of: 
normalizing the said face image into a standard size; 

scanning the said facial component (such as eyes, eyebrows, nose, mouth and 
outline) into a single dimensional array of pixels; 
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subtracting the said single dimensional array of pixels with the average component; 
multiplying the said subtracted single dimensional array of pixels with the said l 81 -' 
order and high-order eigencomponents (such as eigeneyes, eigeneyebrows, 
eigennoses, eigenmouths and eigenoutlines); 

using the resulting component features as description of the component; 
combining the various component features (such as eyes features, eyebrows features, 
nose features, mouth features and outline features) using different attention weights ' 
to describe the face; and 

coding the said features into a coded representation. 

A method for describing a fece image, comprising the steps of: 
normalizing the said fece image into a standard size; 

using a preset block to get the facial component (such as eyes, eyebrows, nose, 
mouth and outline); 

scanning the said facial component (such as eyes, eyebrows, nose, mouth and 
outline) into a single dimensional array of pixels; 

subtracting the said single dimensional array of pixels with the average component; 
multiplying the said subtracted single dimensional array of pixels with the said l rt - 
order and high-order eigencomponents (such as eigeneyes, eigeneyebrows, 
eigennoses, eigenmouths and eigenoutlines); 

using the resulting component features as description of the component; 
combining the various component features (such as eyes features, eyebrows features, 
nose features, mouth features and outline features) using different attention weights ' 
to describe the fece; and 

coding the said features into a coded representation. 

A method for efficiently combining different eigencomponents for fece image 
features description, comprising the steps of: 

using the resulting component features as description of the component; 
combining the various component features (such as eyes features, eyebrows features, 
nose features, mouth features and outline features) using different attention weights ' 
to describe the fece; 

selecting the efficient weights for different components (such as eyes, eyebrows, 
noses, mouth and outline) according to the application field; and 
coding the said features into a coded representation. 

A method for efficiently combining different eigencomponents with whole eigenfece 

for fece image features description, comprising the steps of: 

using the resulting fece eigenfeatures as description of the fece image; 

using the resulting component features as description of the component; 

combining the various component features (such as eyes features, eyebrows features, 

nose features, mouth features and outline features) including whole fece features 

using different attention weights to describe the fece; 

selecting the efficient weights for different components (such as eyes, eyebrows, 
noses, mouth and outline) including whole fece according to the application field; 
and 

coding the said features into a coded representation. 
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3* DETAILED DESCRIPTION OF THE INVENTION 
3.1 Industrial Field of Utilization 



This invention can be used in fece description and recognition for content-based image 
retrieval, human fece identification and verification for bank, security system and 
videophone; surveillance and tracking; digital library and Internet multimedia database. 



3.2 Background and Prior Art 

Human face perception is an active area in the computer vision community. Face recognition 
will play an important role in multimedia database search and many other applications. In 
recent years considerable progress has been made on the problems of fece detection and 
recognition. Different techniques were proposed. Among these, neural networks, elastic 
template matching, Karhunen-Loeve expansion, algebraic moments and isodensity lines are 
typical methods. 

Among these methods, principal component analysis (PCA) or Karhunen-Loeve expansion is 
an important branch. Eigenfece method is derived from PCA and it is convenient to be 
computed and has consistent accuracy in identification. Prior work showed that the PCA 
approach dissociates spontaneously between different types of information. The eigenvectors 
with large eigenvalues capture information that is common to subsets of faces and 
eigenvectors with small eigenvalues capture information specific to individual fece. Hie 
studies show that only the information contained by the eigenvectors with large eigenvalues 
can be generalized to new feces that are not trained. 

3.3 Problem to be solved 

The advantage of eigenfece method is that eigenvectors with large eigenvalues convey 
information relative to the basic shape and structure of die feces. That means features 
extracted from eigenvectors with large eigenvalues can be used to describe major 
characteristics of human feces. However, this is also the weakness of PCA. If we only 
consider the features extracted from eigenvectors with large eigenvalues, we cannot get the 
details of feces which corresponds to individual fece. If these details of individual fece can be 
described with the common features of human feces, the description of human feces can be 
more accurate. 

One drawback of eigenfece method is that the contributions of all fecial component areas are 
same. Instead of evenly distributed on the whole fece, the identity information mainly locates 
at the certain fecial areas, such as eyes, eyebrows, nose, mouth and outline. The check area 
contains less identity information and is relatively sensitive to lighting condition changes and 
fecial expression changes. If the identity significances of fecial components can be used, the 
recognition of human fece can be more accurate. 

3.4 Means of Solving the Problem 
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Eigenface method is effective to extract common face characteristics such as shape and 
structure, In order to get details of feces that are lost when eigenvectors with small 
eigenvalues are truncated, the reconstructed feces with the features from eigenvectors with 
large eigenvalues should be obtained. With the reconstructed face images, the residue images 
between original images and reconstructed matrices can be obtained. These residue feces can 
be looked as high-passed fece images, which still contain rich detailed information for 
individual fece. Tn order to describe these residue feces, eigenfece method can be used on 
these residue feces again. The obtained eigenvectors with large eigenvalues will reveal the 
common characteristics of residue feces. With this method, high-order eigenvectors with 
large eigenvalues can be obtained to extract corresponding features. The combination of these 
features from different order eigenfeces can be used to describe feces effectively. 

Similarly, first order and higher order principal components (eigencomponents) of fecial 
components can be obtained to describe the characteristics of the corresponding fecial areas. 
The combination of these features from different order eigencomponents can be used to 
describe individual fecial components efficiently. Finally, human feces can be represented by 
a combination of different order eigencomponents with different attention weights. For 
different application fields, different components should have different functionalities (its 
strongerness or weakness). Different weights should be assigned for that component. 

The first method for extracting component features for fece description according to the 
invention, comprises the steps of : calculating the 1 st -order eigencomponents with training 
fece images; calculating the 2 nd -order eigencomponents with the said training fece images; 
getting the 2 nd -order component features for a fece image to be described from the 2 nd -order 
eigencomponents; and selecting features to describe the said fece image from the 2 nd -order 
component features. 

The second method for extracting component features for fece description according to the 

invention, comprises the steps of: calculating the I st -order eigencomponents with training 

fece images; getting the l tt -order component features for a fece image to be described from 

the 1 st -order eigencomponents; calculating the 2 nd -order eigencomponents with the said 

training fece images; getting the 2 nd -order component features for the said fece image to be \ 

described from the 2 nd -order eigencomponents; and selecting features to describe the said 

fece image from the 1* -order and 2 nd -order component features. 

The third method for describing a fece image according to the invention, comprises the steps 
of: scanning the said fecial component (such as eyes, eyebrows, nose, mouth and outline) into 
a single dimensional array of pixels; subtracting the said single dimensional array of pixels 
with the average component; multiplying the said subtracted single dimensional array of. 
pixels with the said l rt -ordcr and high-order eigencomponents (such as eigeneyes, 
eigeneyebrows, eigennoses, eigenmouths and eigenoutlines); using fee resulting component 
features as description of the component; and combining the various component features 
(such as eyes features, eyebrows features, nose features, mouth features and outline features) 
using different attention weights to describe the fece. 

The fourth method for describing a fece image according to the invention, comprises the 
steps of: normalizing the said fece image into a standard size; scanning the said fecial 
component (such as eyes, eyebrows, nose, mouth and outline) into a single dimensional array 
of pixels; subtracting the said single dimensional array of pixels with the average component; 
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multiplying the said subtracted single dimensional airay of pixels with the said l rt -order and 
high-order eigencomponents (such as eigeneyes, eigeneyebrows, eigennoses, eigenmouths 
and eigenoutlmes); using the resulting component features as description of the component; 
combining the various component features (such as eyes features, eyebrows features, nose 
features, mouth features and outline features) using different attention weights to describe the 
face; and coding the said features into a coded representation. 

The fifth method for describing a face image according to the invention, comprising the steps 
of: normalizing the said fece image into a standard size; using a preset block to get the fecial 
component (such as eyes, eyebrows, nose, mouth and outline); scanning the said fecial 
component (such as eyes, eyebrows, nose, mouth and outline) into a single dimensional army 
of pixels; subtracting the said single dimensional array of pixels with the average component; 
multiplying the said subtracted single dimensional airay of pixels with the said l*-order and 
Wgh-order eigencomponents (such as eigeneyes, eigeneyebrows, eigennoses, eigenmouths 
and eigenoutlmes); using the resulting component features as description of the component; 
combining the various component features (such as eyes features, eyebrows features, nose 
features, mouth features and outline features) using different attention weights to describe the 
fece; and coding the said features into a coded representation. 

The sixth method for efficiently combining different eigencomponents for fece image 
features description according to the invention, comprises the steps of: using the resulting 
component features as description of the component; combining the various component 
features (such as eyes features, eyebrows features, nose features, mouth features and outline 
features) using different attention weights to describe the fece; selecting the efficient weights 
for different components (such as eyes, eyebrows, noses, mouth and outline) according to the 
application field; and coding the said features into a coded representation. 

The seventh method for efficiently combining different eigencomponents with whole 
eigenfece for fece image features description according to the invention, comprises the steps 
of: usmg the resulting fece eigenfeatures as description of the fece image; using the resulting 
component features as description of the component; combining the various component 
features (such as eyes features, eyebrows features, nose features, mouth features and outline 
features) including whole fece features using different attention weights to describe the fece* 
selecting the efficient weights for different components (such as eyes, eyebrows, noses, 
mouth and outline) including whole fece according to the application field; and coding the 
said features into a coded representation. 



3.5 Operation of the Invention 

The present invention provides a method to interpret human feces which can be used for 
image retrieval (query by fece example), person identification and verification, surveillance 
and tracking, and other fece recognition applications. Li order to describe face characteristics, 
the concept of high-order eigencomponents is proposed according to our observation and 
derivation. At first, all fece images are normalized to a standard size. Then the vertical 
location of eyes is calculated and the fece is shifted to a suitable place. When all these pre- 
processing procedures are finished, the eigencomponents and high-order eigencomponents 
can be derived from a set of training fece images. In order to query a fece image in a fece 
database, the features of the image projected with eigencomponents and high-order 
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eigencomponents can be calculated with the selected eigencomponents and high-order 
eigencomponents. The combination of these features can be used to describe feces. With this 
description, Euclidean distance can be used for similarity measurement In order to improve 
the similarity accuracy, the features should be weighted. 

3.6 Embodiments 

The present invention gives a method to extract higher order eigencomponent features and 
represent a face by combining different order component features. 

With the normalized face images, the eigencomponents and high-order eigencomponents can 
be obtained as follows. 

First use a preset block for the normalized face images to get the fecial component (such as 
eyes, eyebrows, nose, mouth and outline); 

Consider a fecial component ^ which is a one-dimensional vector of raster-scanned fecial 
component, define T as the average component: 

1 M 

a) 

Every fecial component differs from the average component by a vector if* ==0 / — *P. The 
covariance matrix of the data is thus defined as: 

. Q=A 1 ^ (2) 
where -fc"T a ® .X*®]. 

Note that Q has dimension wh x wh where w is the width of the component and h is the 
height The size of this matrix is enormous, but since we only sum up a finite number of 

component vectors M, the rank of this matrix can not exceed M-\ . We note that if V,. (1> is 
the eigenvector of >> (i=l,2,...,W), then A^APy™ - where 

4 (1> are the eigenvalue of A 0)T A 0> , then A^V™ are the eigenvectors of 
A^A^ as we see by multiplying on the left by in the previous equation: 



T 

But A™ y4 (1) is only of size M xM . So defining w/Ve eigenvector of A m A mT we 
have 
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M 

(1) 



0) 



The eigenvalue is the variance along the new coordinate space spanned by 

(l) 

eigenvectors U t . From here on we assume that the order of i is such that the eigenvalues 
A i are decreasing. The eigenvalues are decreasing in exponential feshion. Therefore we 
can project a fecial component r (1) onto only M x « M dimensions by computing 
W W = \» t m } where = u«*r m and 1 < k <J M x . fa ^ *-th coordinate 

of r (,) in the new coordinate system. In this context, W 0) is called 1 st -order component 
features. The vectors U k m are actually images, and are called l*-order eigencomponents Let 
C/ <1) =[«, < V > .-.^ i a> ],ften 

W W = u^pi) (3) 

Since U 0) is an M, xP matrix, we cannot get its inverse. However, we can use its pseudo 

inverse to approximate its inverse. Let U m be the pseudo-inverse of U W , then 
fd) = [/(Dyi) (4) 

where 1 is the reconstructed matrix from 0) and C/ (1) . 

Since a residue fecial component vector still contains rich information for the individual 
component, fecial component features should be extracted from the residue component again. 

Let A™ =lr/ 2) r 2 (2 \.T^ (2) J, X? be the eigenvalues of A™ T A™ and V< 2) be the 
corresponding eigenvectors of A (2)T A i2) . Then A i2)T A m v™ = X™v™ . Based on 
above discussion, the eigenvectors of yl (2) j4 (2)r are v^^A^y™. Therefore we can 
project a residue component r (2) onto only M 2 «M dimensions by computing 

w m =k a> }.«*<=■* 

«tf»- % « r r» (5) 

and 1 ^ A £ . Since U k are the eigenvectors of the residue component, we call #/ 2> 

(2) 

the 2 nd -order eigencomponents and W k the 2 nd -order component features. 
Let C/ (2 > = \u™u 2 (2 \..u Mj {2) j, Eq.(5) can be written , 
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=t/ 2>r r <2) 

=c/ 2 > r |r (1) -c/ 1)+ c/ 1)7 'r <1) ) 

=(lS 2f -If&lfv+l/ 1 ^ (6) 

w*=vfi* (7) 

Since C/ 2 a constant transform matrix and it is just calculated once, it will not affect the 
efficiency of computation. The facial component can be described with 

computational burden does not increase in computing Q(<X>) compared with only computing 
component features from eigencomponents U. 

The residue components are called 2 nd -order residue components and the original 
components are called I s1 -order residue components. 

With the same method, we also can derive 3 rf -order, ^-order and n* -order 
eigencomponents. By projecting the residue components of corresponding order, we can get 
3rd -order, ^-order n* -order component features. With these high-order component 
features, the similarity of components can be defined as the weighted Euclidean distance 
between the projections. Fig.l illustrates the procedure to compute the I th -order 
eigencomponents C/ (,) and the corresponding transform matrix U t . Jh the figure, 
PseudqJnv(B) is the function to calculate the pseudo-inverse of matrix B. 

The measure of dissimilarity of two feces H, and H 2 is defined as a combined distance 
between various fecial component features generated from the projections of 
eigencomponents (i.e. eigeneyes, eigeneyebrows, eigennoses, eigenmouths and 
eigenoutlines) and eigenfeces. 
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If a x = 0 , the similarity of face images will be measured only with 2 nd -order features. 

With the method mentioned above, the method for describing a face image can be considered 
as follows: 

1) Scanning the said fecial component by using a raster scan starting at the top-left comer of 
the component window and finishing at the bottom-right comer of the component 
window into a single dimensional array of pixels; 

2) Subtracting die said single dimensional array of pixels with the average component; 

3) Multiplying the said subtracted single dimensional array of pixels with the said 1 border 
and high-order eigencomponents; 

4) Using the resulting component features as description of the face; 

5) Coding the said features into a coded representation. 

With this method, the human feces can be described effectively and efficiently in space with 
different attention weights corresponding to the significance of identity information of human 
fece. 



3.7 Effects of Invention 

This invention is very effective for describing human feces using component-based features. 
Since the high-order eigencomponents can be calculated only once with the training 
components, the high-order component features can be obtained as efficient as l rt -order 
component features. However, since detailed regional identity information can be revealed 
with high-order component features, the combination of l^-order component features and 
high-order component features of eyes, eyebrows, nose, mouth and outline with different 
attention weights has better face description capability compared with the 1 st -order ejgenfece 
features or combined l*-order and high-order eigenfece features. 
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This invention is very effective and efficient in describe human feces which can be used in 
internet multimedia database retrieval, video editing, digital library, surveillance and tracking 
and other applications using face recognition and verification broadly. 

4. BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1: Shows the procedure for computing 1" -order feature W m 

Figure 2: Procedures for computing i* -order eigencomponents C/ (<) and the corresponding 
transform matrix U t 
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Fig.l 
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Fig. 2 
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ABSTRACT 

This invention provides a component-based method to code and recognize human feces 
effectively and efficiently. High-order eigencomponents are proposed to describe detailed 
regional information of that particular fecial component. A formula is given to calculate high- 
order transform matrix for projection. The high-order component features can be used 
individually to describe a fecial component or combined with 1 st -order component features. 
Since detailed regional identity information can be revealed with high-order component 
features, the combination of l rt -order component features and high-order component features 
of eyes, eyebrows, nose, mouth and outline with different attention weights (based on the 
significance of identity information of the corresponding area) has better face description 
capability compared with the 1 st -order eigenfece features or combined 1 st -order and high- 
order eigen&ce features. 



This invention can be used in Internet multimedia database retrieval, digital library, video 
editing, surveillance and tracking, and other applications using face recognition and 
verification^ 

(Fig. 1) 
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